Debate Publication bias in situ

Background: Publication bias, as typically defined, refers to the decreased likelihood of studies' results being published when they are near the null, not statistically significant, or otherwise "less interesting." But choices about how to analyze the data and which results to report create a publication bias within the published results, a bias I label "publication bias in situ" (PBIS). Discussion: PBIS may create much greater bias in the literature than traditionally defined publication bias (the failure to publish any result from a study). The causes of PBIS are well known, consisting of various decisions about reporting that are influenced by the data. But its impact is not generally appreciated, and very little attention is devoted to it. What attention there is consists largely of rules for statistical analysis that are impractical and do not actually reduce the bias in reported estimates. PBIS cannot be reduced by statistical tools because it is not fundamentally a problem of statistics, but rather of non-statistical choices and plain language interpretations. PBIS should be recognized as a phenomenon worthy of study – it is extremely common and probably has a huge impact on results reported in the literature – and there should be greater systematic efforts to identify and reduce it. The paper presents examples, including results of a recent HIV vaccine trial, that show how easily PBIS can have a large impact on reported results, as well as how there can be no simple answer to it. Summary: PBIS is a major problem, worthy of substantially more attention than it receives. There are ways to reduce the bias, but they are very seldom employed because they are largely unrecognized. Background A study is more likely to appear in the literature, and thus be indexed and publicly available, if it shows a strong, statistically significant, or otherwise "better" result. A study that does not show such results has a greater chance of remaining hidden in a file drawer, either because the author (or funder) does not think the result is worth mentioning, or because journals are less interested in publishing studies that find "nothing". This form of publication bias, wherein readers have access to only a biased sample of the full distribution of results, is well studied, particularly in the literature on meta-analysis and other systematic reviews where the problem is most apparent (a primer on the topic can be found at the Cochrane Collaboration webpage [1]). Even well-read lay people are, at the time of this publication, aware of the problem due to the Published: 05 August 2004 BMC Medical Research Methodology 2004, 4:20 doi:10.1186/1471-2288-4-20 Received: 01 December 2003 Accepted: 05 August 2004 This article is available from: http://www.biomedcentral.com/1471-2288/4/20 © 2004 Phillips; licensee BioMed Central Ltd. This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.